Full-stack documentation: https://docs.openverse.org/_preview/2637

Please note that GitHub pages takes a little time to deploy newly pushed code, if the links above don't work or you see old versions, wait 5 minutes and try again.

You can check the GitHub pages deployment action list to see the current status of the deployments.

Heya, I think the project proposal looks great @zackkrida! Thanks for putting it together! I made very minor comments across the doc

how it relates to this project but also some of its general functionality which I touch on in the proposal.

I think the document has a fair summary of the capabilities of the platform relative to this project. I would just add that the streaming feature of datasets allow for making it accessible to people that may not have access to storage that allows storing the entire data dump. May be helpful as part of the democratisation of this data

Heya, I think the project proposal looks great @zackkrida! Thanks for putting it together! I made very minor comments across the doc

how it relates to this project but also some of its general functionality which I touch on in the proposal.

I think the document has a fair summary of the capabilities of the platform relative to this project. I would just add that the streaming feature of datasets allow for making it accessible to people that may not have access to storage that allows storing the entire dataset. May be helpful as part of the democratisation of this data

+1 on streaming the dataset. It also can allow people to easily and quickly generate various subsets of the data.

Drafting this proposal while I move it into the Revision round. Feedback is still welcomed!

I've addressed reviewer comments and this proposal is now ready for a decision.

I'm going to leave this open for a few more days with the goal of soliciting further community feedback.

I'll leave some scheduling thoughts on the project thread, @sarayourfriend

@apolinario yesterday I saw that the https://huggingface.co/meta-llama/Llama-2-7b model has an access flow which requires accepting terms and signing up through a meta controlled page: https://ai.meta.com/resources/models-and-libraries/llama-downloads/

I am curious: is this type of flow available to datasets, as well? We wouldn't necessarily want folks to wait 1-2 days for access, but the idea of more-explicitly enforcing our terms for dataset usage (respecting proper usage and whatnot) is appealing.

@zackkrida it is possible for datasets: https://huggingface.co/docs/hub/datasets-gated

We could use a small interaction with our Django service (Django rendered page with the form and more flexibility in presentation etc) by using the manual approvals.

As @sarayourfriend noted, gating repos is totally available for datasets!

And it comes in two flavours: manual approval which basically restricts who filled in the form and has the approval done manually (or automated that via a Django service).

BUT if you are going to automate the approvals, the other mode of gated access (automatic approval) would make more sense imo. Basically this requires people to read/accept information they can use it, but everyone that does is accepted automatically and can use it

It is what Mozilla uses for common voice (example here) - I think that could make sense for this project as well

Thanks, @apolinario! I am going to draft this proposal while we make more efforts to consider our specific use conditions and ethical standards for the dataset, along with how they would relate to the "Gating" functionality.

Sounds good! Here's an idea that I had that I hope could help tackling a few challenges that may arise for use conditions & ethical standards:

Multiple-subsets

Instead of a single Openverse dataset (or two, one for visual media and one for audio), we could create multiple subsets based on license or license-grouping, e.g. (not really name suggestion here, just examples):

• openverse/images-cc-by
• openverse/images-cc-by-nc
• openverse/images-cc0
  etc.

All in the same data format, but each could have its own dataset card and its own set of disclaimers and descriptions (and all under your terms of service disclaimers ofc). In one hand, this could make using this datasets for downstream tasks a bit more convoluted/complex (as now one has to engage with/accept terms/process multiple datasets), on the other hand, it would make it very obvious what each dataset could be for, and it could inform downstream users very specifically what they are doing, as they would need to write code that looks like:

from datasets import load_dataset
dataset_cc_by = load_dataset("openverse/images-cc-by", token=True)
dataset_cc_by_nc = load_dataset("openverse/images-cc-by-nc", token=True)
#do downstream tasks processing both

That could make it pretty clear that they should not do that if they are looking into doing smth commercial. Although filtering a column by value (e.g.: the license column) enough with the HF datasets library, but maybe this is a way to make it even more explicit and understood even from reading the code - and not only by reading the model card.

This could also co-exist with gating, so each dataset repo could be gated. Btw the load_dataset function gives a 403 error if the dataset is gated and your HF user isn't in.

Merging this for now so the document is in our documentation site, even though we are not planning on pursuing it at this time.